Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support the parallel conversion from ZeRO checkpoints to FP32/FP16/BF16 param weight #6655

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Support the parallel conversion from ZeRO checkpoints to FP32/FP16/BF16 param weight #6655

wants to merge 7 commits into from
+139 −55

Conversation

xylian86
Copy link
Contributor

@xylian86 xylian86 commented Oct 23, 2024 •
edited
Loading

This PR adds parallel conversion support for the zero_to_fp32 script using universal checkpoints, addressing the request at ISSUE#6526. "The previous version was written 3 years ago models were small and converted fast. Now with 70B+ models the conversion can take hours."

  • Switch to universal checkpoint API
  • Support Frozen Parameters
  • Support Shared Parameters
  • Add the support for output to SateTensor
  • Add the support for output to FP16/BF16

@xylian86 xylian86 marked this pull request as ready for review October 23, 2024 15:22
@loadams
Copy link
Contributor

loadams commented Oct 25, 2024

Hi @xylian86 thanks for the contribution, could you run the pre-commit formatter locally to fix the Formatting error?

@loadams loadams requested review from tohtana and removed request for awan-10 October 25, 2024 01:57
@xylian86
Copy link
Contributor Author

@loadams Thank you for the reminder. I run the pre-commit formatter locally before the PR but It seems that formatting outputs can vary across different environments. After switching to a new machine with ubuntu-22.04 and re-running pre-commit formatter, I've resolved the formatting issues.

@loadams
Copy link
Contributor

loadams commented Nov 25, 2024

@loadams Thank you for the reminder. I run the pre-commit formatter locally before the PR but It seems that formatting outputs can vary across different environments. After switching to a new machine with ubuntu-22.04 and re-running pre-commit formatter, I've resolved the formatting issues.

Hi @xylian86 - looks like there are a few merge conflicts now - could you resolve those, then we can get this merged?

@loadams
Copy link
Contributor

loadams commented Jan 7, 2025

@loadams Thank you for the reminder. I run the pre-commit formatter locally before the PR but It seems that formatting outputs can vary across different environments. After switching to a new machine with ubuntu-22.04 and re-running pre-commit formatter, I've resolved the formatting issues.

Hi @xylian86 - looks like there are a few merge conflicts now - could you resolve those, then we can get this merged?

@xylian86 - ping on this if you have time to resolve the merge conflicts we can get this reviewed as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tjruwase tjruwase Awaiting requested review from tjruwase tjruwase is a code owner

@tohtana tohtana Awaiting requested review from tohtana tohtana is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xylian86 @loadams @tjruwase